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Abstract. We conjecture a geometrical form of the Paley-Wiener theorem 
for the Dunkl transform and prove three instances thereof, by using a reduction 
to the one-dimensional even case, shift operators, and a limit transition from 
Opdam's results for the graded Hecke algebra, respectively. These Paley- 
Wiener theorems are used to extend Dunkl's intertwining operator to arbitrary 
smooth functions. 

Furthermore, the connection between Dunkl operators and the Cartan mo- 
tion group is established. It is shown how the algebra of radial parts of in- 
variant differential operators can be described explicitly in terms of Dunkl 
operators. This description implies that the generalized Bessel functions coin- 
cide with the spherical functions. In this context of the Cartan motion group, 
the restriction of Dunkl's intertwining operator to the invariants can be inter- 
preted in terms of the Abel transform. We also show that, for certain values 
of the multiplicities of the restricted roots, the Abel transform is essentially 
inverted by a differential operator. 



1. Introduction and overview 

In recent times the study of special functions associated with root systems has 
developed to a considerable degree. Starting with a number of conjectures by Mac- 
donald, and the work of Heckman and Opdam on multivariable hypergeometric 
functions in the late 1980's, the development of the theory was greatly enhanced 
by the introduction of rational Dunkl operators by Dunkl 6 . Through various 
intermediate steps of generalization, these operators can even be said to have ulti- 
mately provided crucial building blocks for Cherednik's work on double affine Hecke 
algebras and the q-Macdonald conjectures. 

Originally, before the introduction of Dunkl operators, the idea when studying 
special functions related to root systems was to consider root multiplicities in the 
theory of spherical functions on Lie groups as parameters, and then to develop a 
theory for Weyl group invariant objects in this more general situation, without the 
aid of the presence of the group. It was this point of view which underlay the 
Macdonald conjectures and which led Heckman and Opdam to the development 
of their theory of hypergeometric functions in higher dimension. One of the main 
technical problems in this context is the description of the generalized radial parts of 
invariant differential operators. Apart from an explicit formula for the generalized 
radial part of the Laplacian — an expression which was in fact the starting point for 
Heckman and Opdam — the other operators remain somewhat intangible. 
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This problem disappeared when Dunkl found the operators which have come to 
bear his name. They are parameterized deformations of the ordinary derivatives, in- 
volving a finite reflection group, for which it is still relatively easy to study the spec- 
tral problem and develop the theory of the corresponding Fourier transformation — 
the Dunkl transform. The invariant part of the theory then answers questions in 
a generalized theory of spherical functions — for the Cartan motion group, to be 
precise — as described above. The operators under consideration being explicitly 
given, it is actually easier to study the general context of Dunkl operators, and 
then specialize to invariant objects later on, than it is to restrict oneself to the 
invariants from the outset. The same holds true for the modification of Dunkl op- 
erators into Chcrednik operators, which gives rise to a representation of the graded 
Hecke algebra 27 and the non-invariant "envelope" of the work of Heckman and 
Opdam on hypergeometric functions. 

Quite remarkably, the theory of spherical functions in analysis on Lie groups 
can in a number of situations thus be regarded as only the invariant part of a 
general non-invariant theory for Dunkl-type operators. At the time of writing, it is 
unknown to the author whether there is an underlying reason for this phenomenon. 

In view of all this, these Dunkl operators and their modifications have attracted 
considerable attention in various areas of mathematics and mathematical physics 
during the last decade. To get an impression of their influence on the development 
of special functions associated with root systems — also in the general non-invariant 
context — we refer to, e.g., [El, [2H|i QH], PII]' For their use in the study of integrable 
quantum many body systems of Calogero-Moser-Sutherland type we refer to ^ 
and the bibliography therein. 

In this paper, we are mainly concerned with the further development of the 
general theory of the Dunkl transform, notably with the Paley- Wiener theorem. In 
addition, we describe the relation between Dunkl operators and the Cartan motion 
group. We will now turn to an overview of the contents of this paper with regard 
to these two subjects. 

The first substantial results for the Dunkl transform, i.e., the Plancherel theorem 
and inversion theorem, were obtained by Dunkl ,9, and the author |21|. Two Paley- 
Wiener theorems were established in (521 Chapter 3] (unpublished), of which this 
paper is an extension. Whereas the proof of the inversion theorem in j21j is partly 
a formal argument, based on various symmetry properties of the eigenfunctions of 
Dunkl's operators, it will become apparent below that quite some more work is 
required in Paley- Wiener theory, due to the lack of adequate asymptotic results in 
the spectral domain. We will establish three Paley- Wiener theorems, each of these 
being a special case of a conjectured geometrically more precise general Paley- 
Wiener theorem, which can be found below as Coniecture l4.1l Each version requires 
a different technique. The first and most general version relies on a reduction to 
the one-dimensional even case — where asymptotic results are available — and some 
non-trivial results from general representation theory and from the representation 
theory of the orthogonal group. The second version, which is proved for a discrete 
set of parameters only, is established through shift operators. Finally, the third 
version, which we prove for Weyl groups, follows from a limit transition from results 
of Opdam 27 . This limit transition has some interest in itself, and for a discrete 
set of parameters its validity has also been established by Ben Said and 0rsted, 

cf. El. 
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Turning to the Cartan motion group, for which the material is taken from |22j . 
we will show how the algebra of radial parts of invariant differential operators can 
be described explicitly in terms of Dunkl operators with suitable values of the 
parameters. From this description it is easy to see that the invariant components 
of the eigenfunctions of the Dunkl operators are precisely the spherical functions. 
We can then also describe Dunkl's intertwiner operator, or rather its restriction 
to the invariants, in terms of the Abel transform. For certain multiplicities, the 
results on shift operators imply that this Abel transform is essentially inverted by 
a differential operator. 

The organization of this paper is as follows. In Section|2we establish the neces- 
sary general definitions and notations, and we recall some previous results. Sectional 
contains a number of useful formulas. We do not claim any originality in particular 
for these formulas, but we believe that the proofs are considerably simpler than 
the ones in the existing literature. The conjectured Paley-Wiener theorem can be 
found in Section ^ as can the proven three instances of it which were described 
above. These Paley-Wiener theorems are then used in SectionElto extend Dunkl's 
intertwining operator to the smooth frmctions. To conclude with. Section con- 
tains the details of the connection between Dimkl operators and the Cartan motion 
group, including the results on the Abel transform. 

2. Notations and previous results 

Let a be a real vector space of finite dimension N which is equipped with an 
inner product ( . , . ) , inducing a Lebesgue measure dx on a. We let ac = a (8)m C 
denote the complexification of a and we extend the form ( . , . ) to a bilinear form 
on ac, again denoted by ( . , . ). Both a and ac can be identified with their duals via 
(.,.); for ^ in a or ac the corresponding linear functional is then denoted by ^* . 
Define the orthogonal group 0{N) — 0{a, { . , .)). The norm | . | which is induced 
on a by ( . , . ) extends to an 0(iV)-invariant norm on ac, also denoted by | . |. There 
is a natural action of 0{N) on functions: 

Let G C O(A^) be a finite (real) reflection group with corresponding root system 
R. We may and will assume that (a, a) = 2 for all a G R. U a G R, then is 
the orthogonal reflection in the hyperplane perpendicular to a. We choose and fix 
a positive system in R. 

A function k : R <C is called a multiplicity function if k is G-invariant. We 
write /c > if all values of k are non-negative, with analogous notations Re fc > 
and fc > 0. We let K denote the vector space of multiplicity functions on R. 

Let k G K and ^ G a. Then the corresponding Dunkl operator T^{k) is defined 

by 

Here G a) is the (unnormalized) directional derivative operator; we have used 
the notation Mf for pointwise multiplication by a function /. 

The definition of the operators is independent of the choice of the positive sys- 
tem. The T^{k) leave the polynomials V invariant, mapping the homogeneous 
polynomials Vn of degree n into Vn-i- Furthermore, the spaces C°°{a) of smooth 
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functions, 'D(a) of compactly supported smooth functions and 5(a) of rapidly de- 
creasing smooth functions are also invariant |^ Lemma 2.1]. 

Quite remarkably, the T^{k) form a commutative family: T(^{k)T^(k) = Trj{k)T^{k) 
for all ^, 77 g a and all A: g if , as was proved by Dunkl '5', see [TS] for a different 
proof. As a consequence of the commutativity the map ^ 1— s- T^{k) extends to V; 
for p £ V the corresponding operator is then denoted by Tp{k). The important 
fc-Laplacian T\ \2{k) is denoted by A^. 

For fc > 0, Dunkl [Jj has constructed a linear isomorphism Vk : V 1^ V, homoge- 
neous of degree 0, such that V^l = 1 and T^{k)Vk — Vkd^ (^ € a). An alternative 
approach for more general k can be found in (T5i |. A concrete description of this 
intertwiner operator Vk is presently still unknown, with the exception of the one- 
dimensional case and the case A2 QSk . Significant abstract results were obtained 
by Rosier who showed, amongst others, that Vk is for fc > a positive operator 
which can be described in terms of measures. 

Let A € ac and consider the simultaneous eigenfunction problem 

(2.1) T^ik)f=i\,m (eea). 

This problem was studied first by Dunkl |S] for fc > 0; later Opdam [32] treated the 
general case. One more definition is needed to state the general result: a multiplicity 
function fc G if is said to be singular if the simultaneous kernel of {Tj(fc)}^ga in V 
is non-trivial, i.e., if it properly contains the constants. We will use the self-evident 
notations JsT^'^'s and if'^'s. 

The set if'^'^s j-^^s been determined in all cases ^] and some partial informa- 
tion about the simultaneous kernel for singular multiplicities can also be found in 
[loc.cit.]. The general nature of the simultaneous kernel for singular multiplicities 
is, however, still unknown, with the exception of the case An of the symmetric 
groups which has been solved by Dunkl 11, 12 . In this paper we will mainly be 
concerned with multiplicities satisfying Refc > 0. Such multiplicities are regular, as 
is most easily seen by considering the operator X]t=i s*^ei (k) for an orthonormal 
basis {ei, . . . , cat} of a 

For regular multiplicities, the result from 26 for the eigenfunction problem is 
as follows. 

Theorem 2.1. For allk £ K^°s iJi^ eigenfunction problem (12.1(1 has a 1- dimensional 
solution space for all A G ac. This space contains a (unique) function F,xjpq[X, fc, . ) 
such t/iai Expq(A, fc, 0) = 1. Furthermore, Exp(3(A, fc, .) extends to a holomorphic 
function on oc, and 

Expc(. , . , .) : ac C 

is a meromorphic function with poles precisely in K'^™^. 

In order to be able to define the corresponding Fourier transform — the Dunkl 
transform — for sufficiently general functions, one needs non-trivial bounds for the 
eigenf unctions. It is shown in [25 that for Re fc > one has 

(2.2) |Expq(A, fc,2)| < y|G[exp(maxRe(5A, 2:)) (A, z G ac), 
in particular 

(2.3) |ExpG.(iA,fc,x)| < (A,xGa). 
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If A: > then the constant \/|G| in (|2.2|) and H2.3|) can in fact be improved to 1, as 
a consequence of Rosler's results on the intertwiner operator |29| . 

Following Dunkl, we define for Re fc > the G-invariant complex- valued weight 
function Wk — YiaeR lo^*!^*^"- The T^{k) are anti-symmetric with respect to this 
weight function: if / G S{a) and g is smooth such that both g and T^{k)g are of at 
most polynomial growth, then 

(2.4) f {T^{k)f)gwu dx = - f f {T^{k)g)wk dx. 

J a J a 

In view of H2.3|) , the Dunkl transform for Re A; > is meaningfully defined on 
ii(a, \wk{x)\dx) as 

DkfW = — [ f{x)ExpQ{-iX,k,x)wkix)dx (A G a, f £ Li{a,\wk{x)\dx)). 
Its alleged inverse is 

Ekf{x) = — I f{X)Expc;{iX,k,x)wk{X)dX (x G a, f £ Li{a,\wk{X)\dX)). 

Cfc Ja 

Here the normalizing constant Ck is the Mehta integral, which is defined as 

Ck = tpwk dx. 

J a 

There is a closed expression for this integral; this former Macdonald conjecture 
has been proved by Opdam, first for Weyl groups [21] and later for finite reflection 
groups in general For our needs it suffices to know that c^. 7^ if Re fc > 0, 
which can be proved by more elementary means |21l Corollary 4.17]. 

The first results for the transform, and notably a version of the Plancherel theo- 
rem, were obtained by Dunkl j9j. Later on, a more systematic study was undertaken 
in [211 . The main properties and results are as follows. 

Theorem 2.2. Let Refc > and ^ £ a. Then: 

(1) Dkmk)f = M,^,Dkf if £ 5(a)). 

(2) Ekmk)f = -M,^,Ekf if £ 5(a)). 

(3) DkM^^.f - -T^{k)Dkf if £ 5(a)). 

(4) EkM^^.f^ T^{k)Ekf if £ 5(a)). 

(5) Dk is a linear homeomorphism o/5(a), with inverse Ek- 

(6) /// G Li(a, \wkix)\dx) andDkf £ Li(a, \wkix)\dx) thenDkEkf EkDkf^ 
f a.e. 

(7) Ifk > thenDk maps Li{a,Wk{x) dx)nL2(a,Wk{x) dx) into L2{a,Wk(x) dx) , 
isometrically with respect to the two-norm corresponding to Wk{x)dx, and 
extends uniquely from Li(a, Wk(x) dx) n ^2(0, Wk{x) dx) to a unitary oper- 
ator on L2{a, Wk{x) dx). 

Remark 2.3. The definition of Dunkl operators can be generalized to complex 
reflection groups 

3. FORMULARIUM 

In this section we establish some useful formulas. Some of them are known from 
the work of Dunkl, who based his proofs to a large extent on the existence of a 
fc-harmonic decomposition for polynomials Theorem 1.7]. However, with the 
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benefit of hindsight we can simpUfy some of the original proofs considerably, by 
systematically exploiting the following commutator relation ^ Proposition 2.2]: 



(3.1) 



= -mk) (Cea). 



To start, we first note that Aj, : V ^-^ V is homogeneous of degree —2, hence 
locally nilpotent. This enables us to define the linear automorphisms e 2 of 7^. 
These automorphisms will play an important part in what follows, together with 
the Gaussian ^p{x) ~ exp(— |a;p/2). It is known [2] that the Gaussian is an 
eigenfunction of both Dk and Ek with eigenvalue 1. 

Lemma 3.1. Let ^ G a. Then for arbitrary k we have 



(1) inEndc(7').- 



Me , e- 



= T^{k) o e 

(2) in Homc('P,5(a)).- T^ik) o o e"^ 



— —M^ o e ~ o M^, . 



Proof. The first part follows immediately from H3.1|l and the obvious fact that 
[Afe, T5(fc)] = 0. As to the second part, note that T(^{k) o = o - o 

Mj. , as a consequence of the G-invariance of -ip. Hence 



T^{k) o o e 



= M^oT^{k) oe-— ~ 
= M^o T^{k) o e^T^ - 
= M^o T^{k) o e^T^ - 
= -M^ o o M{.. 



o Mf* o e 



o 



o 



{ [^^e 



e 20 



oe 2-fe 20 M, 



} 



□ 



Repeated application of the second part of Lemma l3 . II vields the formula 

(3.2) Tp(fc) {(e-^q) V} = (-l)'i<=8P {e-^(M)}^, 

for homogeneous p € V and arbitrary q & V . Taking q — 1 this implies, together 
with Theorem 12. 21 the following result, which is equivalent to |51 Proposition 2.1]. 

Corollary 3.2. If p Cz V is homogeneous and Re/c > 0, then 

(1) Dkip^) = (-i)d-gP (e-%p) V- 

(2) i;fc(pV) =*''°^^(e-^p)^. 

The second part of Lemma 13.11 also enables us to reprove the symmetry of a 
bilinear form which was introduced by Dunkl 8 , as follows. For p,q e V, put 
{p,q)k = {Tp(k)q) (0). Although it is not obvious from the definition, this bilin- 
ear form on V is actually symmetric. This symmetry follows from the following 
generalization by Dunkl [loc.cit.] of a result of Macdonald: 



(3.3) {p,q)k^^J (e "^^"p^le '^^^ tjjWk 



dx (Re/c > 0, G -P), 
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in which the right hand side is obviously symmetric. In order to re-estabhsh H3.3|l . 
denote the right hand side by [p, q]k- Now (|2.4|l and Lemma FH. II imolv that 

1 



[P,mk)q]k = 



2 pj (r^(fc)e 2 qj^uukdx 
(fc) Tp^ (^e^^q^ Wkdx 

^(Mep)}(e 



q ] tpWk dx 



= - I -T> 

Ck 

1 

Ck 

= [M^.p,q]k- 

But the other form (. , .)k also has this property: {M^*p,q)k — {p,T^{k)q)k, as a 
direct consequence of its definition. Since it is easy to see that {l,q)k ~ [Ij 9]fej an 
induction with respect to degp then proves that {p,q)k — [p , q]k, which is (|3.3|) . 

The following proposition will be used in the reduction of the proof of the Paley- 
Wiener theorem 14. 101 to the one-dimensional even case. 

Proposition 3.3. //Refc, Refc' > 0, then 

Ek'Dkip^) = (e^^e^p^tP {peV). 
Proof. We may assume that p is homogeneous. Then CoroUarv 13.21 implies that 

Y- 

Z-^ 77 ,1 



Ek'Dk (pV) - i-if'^^PEk. 

= (e 2 e 



2 



~ jdcgp-2« 

> j e 2 

^ n! 

.71=0 



2 



p] ip. 



□ 



To conclude this section, we re-establish a formula of Heckman T6' which ex- 
presses the pivotal role of the fc-Laplacian Afe: 



(3.4) 



1 



ad ■ 



A. 



Mr, 



for p £ Vn- The proof in [loc.cit.] is based on representation theory for sl(2), but it 
can also be seen directly, as follows. We may assume that Mp — n"=i -^'^c*- Then 



ad — 
2 



E 



Jl! ■ • - in! 



ad — 



ad — 



But (ad Afe)^ M{* = as a consequence of (I3.1|l and the commutativity of the T^{k), 
so the only surviving term in the summation is the one with ji = . . . — jn = ^■ 
Using H3.1|l once more, this proves H3.4|l . 



4. PALEY-WIENER THEOREMS 



In this section, we conjecture a geometrical form of the Paley- Wiener theorem 
for the Dunkl transform, and present several theorems to support it. 
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Establishing notation, for 5 C a we let 'D{S) denote the smooth compactly 
supported functions with support contained in S. If S is compact and non-empty, 
then we define the indicator : o i-^ M as Is{x) = maxyizs{x,y) for x a. For 
such S, let TCs be the functions on ac of Paley- Wiener type corresponding to S, 
i.e., those entire functions with the property that for each integer M > there 
exists a constant jm such that |/(A)| < 7a/(1 + exp /5(Im A) for all A G ac- 

We then conjecture the following. 

Conjecture 4.1 (Paley-Wiener conjecture). Let G be a finite reflection group. If 
Refc > 0, and if S is a non-empty G-invariant compact convex subset of a, then 
Dk is a linear isomorphism between 'D{S) and Us- 

There is some evidence supporting this conjecture: 

• The inversion theorem and |21l Corollary 4.10] show that Dk is an injective 
map from 2?(S') into Hs- 

• For fc = the statement holds |^ Theorem 7.3.1]. 

• If 5 is a ball centered at the origin, then a reduction to the one-dimensional 
even case, where asymptotics can be used, enables us to establish the state- 
ment as Theorem 14. 101 below. 

• If the ka are all strictly positive integers, then the statement can be estab- 
lished, using shift operators, as Theorem 14.111 below . 

• If G is a Weyl group, and if S is the intersection of convex hulls of orbits, 
then a limit transition from results of Opdam establishes the statement as 
Theorem 14.151 below. 

The main obstacle for a possible proof of the conjecture along the usual lines, 
using a contour shift, is the absence of adequate asymptotic results for the Dunkl 
kernel. There are some asymptotic results available [2], but these fall far short of 
what is needed. It is to be expected that better results could be obtained if more 
were known about Hosier's representing measures PHli but as yet these remain 
elusive. But even if much stronger asymptotic results became available, the proof 
of Theorem l4.4l below seems to suggest that additional monodromy arguments may 
then still be necessary. 

Remark 4.2. In jSH], a proof of Conjecture 14.11 if A; > and J2aeR+ > is 
presented. That proof, however, is not correct and to our knowledge Conjecture 4.1 
is at the time of writing still open. 

4.1. The case of arbitrary G and Refc > 0. Throughout this section, Bji will 
denote the closed ball in a with radius R and the origin as center. The space 'D{Bji) 
carries the usual Frechet topology of uniform convergence of all derivatives. 

Our approach of the Paley-Wiener theorem for such sets — under the assumption 
that Re fc > — consists of three steps, cf. the proof of Theorem l4. 101 First, we prove 
the result for even functions in one dimension. Second, it is shown that this implies 
the theorem for radial functions in arbitrary dimension. In the third step we finally 
prove that the result for radial functions implies the theorem for general smooth 
functions with support in a closed ball Bn. 

Remark 4.3. The special role of radial functions in the theory has been noted 
by several authors [221 QOI [HHIi e.g., the Dunkl transform of a radial function 
is again radial [22]. In Paley-Wiener theory this phenomenon is encountered once 
more, when reducing the radial case in arbitrary dimension to the even case in one 
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dimension. The one-dimensional even case can be handled by either Weyl fractional 
integral operators [HH|) or as in our approach below, which uses classical results for 
the asymptotics of Bessel functions combined with a contour shift. The final step 
in our approach, from the radial to the general case, and for which a reference to 
[22] was given in is in a sense the deepest, since it ultimately rests on the 
general theory of representations of compact groups in Frechet spaces. Recently, 
an alternative reduction to the one-dimensional case has been introduced which 
circumvents the inference of the general case from the radial one |35| . This reduction 
is based on results for /c-harmonic polynomials and for functions supported in Bj^ 
it gives a proof of the Paley- Wiener theorem which is independent of the results in 
the present paper. 

Starting with the proof, the one-dimensional even case is settled in the following 
theorem, where, as usual, invariance is denoted by superscripts. 

Theorem 4.4. Let a be one- dimensional and Refc > 0. // / G Ti.^ , then Ekf S 

Proof. Let / be as in the statement. Then obviously Ekf is an even rapidly de- 
creasing smooth function. We have to show that 

/(A)Exp2^(a,A;,a;)|Apfc d\ = 

if a; > i?. Fix such x. From ^9^ we have 

Expz^(zA,fc,x) = r(fc + i) (^^y {jfc_i(Aa;)-HzJfe+i(Ax)}. 
Using the invariance of / we see that we are left with showing that 

poo ^ 

(4.1) / /(A)(Ax)^-'=4_i(Ax)|Ap''^dA = 0. 



This expression makes it obvious that there are two obstructions for a direct appli- 
cation of the classical argument of shifting the contour to infinity. First, the Bessel 
function has exponential growth in both the positive and negative imaginary di- 
rections, and second the weight function jAp'^ is in general not the restriction of a 
holomorphic function on the upper or lower half plane. So we proceed indirectly. 
Recall the definition of the Bessel functions of the third kind: 



i sin VK 

—ismuTT ' 



so that 

2 • 

If V is an integer then a limit has to be taken. In our case, this occurs if k is 
half-integer, but by continuity in k we may and will assume that this is not the 
case. 
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For our purposes, the important property of these functions is the asymptotic 
behaviour 9.2.7]: 

(4.2) hI^Hz) = (^l + O 

vahd if — TT < arg z < 2t: (which is to be interpreted in the sense of analytic 
continuation). Note that the range of vahdity of this asymptotic development 
contains the entire upper half plane and that (in contrast to the ordinary Bessel 
function) this Hankel function has exponential decrease in the positive imaginary 
direction. 

Define 0(1), 0(2) • (o,oo) ^Chy 

0(1) (A) = (Ax)5-fcii-W,(Aa;)A2'= 

and 

0(2)(A) = (Aa;)5-'=i/p_\(Ax)A2'=. 

Let ^c^'' denote the analytic continuation of 0(i) from (0, oo) to C\ {it \ t < 0}. If 
one recalls that Ju{z) = z'^ J^iz) with J^, entire and Z2-invariant, one notes that 
0c^-'(A) remains bounded as A ^ in C \ {it | t < 0}, since Re/c > 0. A small 
computation will also make it clear that 0c^''(A) — 0(2) (—A) for A < 0. 

Using the invariance of / and the weight function, we then compute as follows. 

/(A) (Aa;)^-'=J,_.(Aa;)|A|2'= dX = 

oo ^ 

/(A) (Ax)^-'4_i(Aa:)A2'= d\ 
/(A) (Aa;)^-*'' {h^^\_{\x) + i/fji (Aa;)) dX 
/(A) (0(1) (A) + 0(2) (A)) dX 



= lim|y /(A)0(i)(A)dA + y /(A)0(2)(A)dA 
= lim(/ /(A)0(i)(A)dA+ / /(-A)0(2)(-A)dA 

Ue J-oo 

= lim|^"/(A)0W(A)dA + J '/(A)0W(A)dA 

Now /0c^'' is holomorphic on <C\{it \ t < 0}, and, since a; > i?, it has exponential 
decrease in the positive imaginary direction, as a consequence of 1)4. 2|l . The classical 
argument therefore establishes H4.1|l . with a minor modification involving a semi- 
circle of radius e around in the upper half plane and using that /0i is bounded 
around 0. □ 

Next, we proceed with the reduction of the general radial case to the one- 
dimensional even case. To this end, fix a; o 7^ in a, and define the map ReSa,;, : 
iS(a)*-'(^) I— > 5(R)^^ by restricting to the line passing through a;o: 



(Res.„ /)(.) = / ( (. e R, / e 5(a)0(^: 
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The following proposition, which implies the Paley- Wiener theorem for radial 
fmictions (as will become apparent in the proof of Theorem 14. 10(1 . involves the 
Dunkl transform for general a and R both at the same time. We therefore add a 
subscript Z2 in the latter case for clarity. Let iS(a)'-'''^' denote the 0(iV)-invariants 
in 5(a). 

Proposition 4.5. Suppose Refc > 0. Let / G 5(a)'^'^'' and put 7 = X)aei?+ ^<^- 
Then 

for all nonzero Xq e a. 

Proof. Let So{a) — {^"0 \ P & 'P}, with ip denoting the Gaussian as in Section |31 
It is known that So{a) is dense in 5(a), see |^ p. 263] for this particular result, 
or for a general framework for this type of problems. Note that the canonical 
projection from 5(a) onto S{a)'^^-^^ is continuous as a consequence of the closed 
graph theorem, implying that 5o(a)'^'^^' is dense in S{a)'^''^\ It is therefore, 
by linearity and continuity, sufficient to prove the proposition for a function / 
of the form jxp'?/;, where g is a non-negative integer. This can be done using 
Proposition 13.31 and an identity for Laguerre polynomials. Recall the definition: 

^-^ ml \n — m 
Then the following identity holds P 22.12.6]: 

n 

(4.3) Li:Hx)Li%^{y) = li^+^'+'H^ + v)- 

It is known 8 , Proposition 3.9] thate^|a;p9 = 2'iq\L[^^'^~'^^ (-|a:|V2) ande~^\x\^'i = 
(-2)«g!L^^/^+''~^^ {\x\^/2). Using Proposition Owe therefore find 

E,Do{\x\^^ij) = {e-^e^\x\^i^i, 

m=0 \ ^ / 

The special case n = q, y = 0, a = N/2 + 7 - 1 and f3 = -N/2 - g of (|4.3|l then 
shows that 

EkDo (Ixl^^^) = (-2)V4^"^^ (l^lV2) V-. 
Curiously enough, the dimension N has dropped out, and the proposition follows 
immediately from this observation. □ 

It follows easily from this result that the Dunkl transform of a radial function is 
again radial, retrieving this result from | 32| . 

Let Vq{Br) = Span{Tp(0)/ \ p G P, f e I?(Bfl,)0(^)}. The following density 
result is crucial in the step from the radial to the general case. 

Proposition 4.6. 'Do{Bji) is dense in V^Bb). 



12 



MARCEL DE JEU 



This proposition implies a special case of 19, Cor. 7.8, p. 310], but the latter 
result does not seem to imply the proposition. The proof of ProDOsition l4.6l is based 
on representation theory for compact groups in general and 0{N) in particular; we 
recall a few facts to start with. 

Let n : K 1—^ Aut(£') be a strongly continuous representation of the compact 
group K in the automorphism group of a Frechet space E. A vector e d E is K- 
finite if the orbit of e under K spans a finite dimensional subspace of E. If S € K 
(the unitary dual of K), then a vector e e ii' is K -finite of type S if the span of the 
orbit of e decomposes into finitely many copies of 6. For 6 G K, let Es denote the 
subspace of K-&mte vectors of type 6. By |18[ Lemma 1.9, p. 396] the subspace 
0^g^ Es of i^-finite vectors is dense in E. 

The natural action of O(A^) on 'D{Bi^) is such a strongly continuous representa- 
tion. Hence Proposition 14.61 will be implied by the density of the i4'-finite vectors, 
as mentioned above, once we have proved the following result. 

Proposition 4.7. 'Do{Bji) is the space of 0{N)- finite vectors in'D{B]^). 

Obviously, any element of T>o{Bfj) is 0(A^)-finite, but the reverse appears to be 
harder. The proof of Proposition l4. 71 below uses some Fourier analysis; for clarity 
of notation, we will temporarily denote the ordinary Fourier transform by rather 
than Do- We need the following classical result in the representation theory of 



Theorem 4.8. Let 7i„ be the harmonic polynomials in Vn- Any p € Vn has a 

unique decomposition of the form 



with pi G 'Hn-21- The map 7r„^; : Vn 'Hn-21 that sends p to pi is 0{N)- 
equivariant. Furthermore, the representations of 0{N) on the spaces 7i„ (n = 
0, 1, . . . ) are irreducible and mutually inequivalent. 

As a further preparation for the proof of Proposition 14. 71 we need the following 
result. 

Lemma 4.9. Let S € 0{N) and f G 'D{Bii). Suppose that the orbit of f un- 
der 0{N) spans a copy Vg of 5 in T>{Bfi). Then there exist a harmonic homoge- 
neous polynomial p of type 6 and scalars Am (rn = 0, 1, . . . ), such that T{f){z) = 
X]m=o '^'"('^' '^)'" (-^ ^ 'C'^), where the series converges uniformly on compact 
subsets o/C^. 

Proof. By the Paley- Wiener theorem the Fourier transform J-{f) is an entire func- 
tion on C^. J- commutes with the 0(A^)-action, so there is a copy of S in the space 
of functions of Paley- Wiener type. Since the action of 0{N) on entire functions is 
homogeneous (in the sense that the action is on each homogeneous component of 
the power series development around separately), we conclude that S also occurs 
in V. By Theorem 14 . 81 there exists a unique copy, Ti.i say, of S as the harmonics of 
some homogeneous degree. 

For n > 0, let 7r„.; denote the map Vn 'Hi defined by the decomposition in 
Theorem 14.81 Also, for n > and g € V^, let J-n{g) denote the component of 
homogeneous degree n in the power series development of T{g) around 0. Consider 



0{N). 
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the 0(A^)-equivariant maps 7r„^; o Tn V5 t-^ Hi {n > 0). Since J-{f) 7^ 0, there 
exists 710 such that 7r„gj o JF„^ is non-zero, hence an isomorphism. But then for 
all n the map (7r„.i o J^^) o (7r„g_i o J-ng)^^ : 7i; i— > 7i/ must be equal to a multiple 
Xn of the identity. Hence f{z) = J2m=o ^m{z, z)™-p{z) [z e C^) for some nonzero 
p ^ Hi- Since there exist z G with p{z) ^ and \z\ arbitrarily large, the factor 
p{z) can be brought outside the summation. □ 

Proof of Proposition |^ . 7| As remarked before, 'Do[Bj^) is certainly contained in the 
space of 0(A^)-fLnite vectors in 'D{Bj^). Conversely, let / G ^{Bb) be an 0(A^)- 
finite vector. We may assume that the orbit of / spans a copy of 5 in 'D{Bii) for 
some 6 £ 0{N). Applying Lemma [4.91 one obtains a factorization J-{f) = ph of 
J^{f) in a polynomial p and an entire 0(iV)-invariant function h. We proceed to 
show that h is in fact in Ti-Ba- ESI Lemma 8.3] there exists a constant A > 
such that 

(4.4) \h{z)\<Af \T{f){z + w)\daN{w) (z S C^), 

JrpN 

where = {(e^'^i, . . . , e*^") | 6*^ G R (j = 1, . . . , N)} is the iV-torus and (Tn is the 
normalized Haar measure on . There exist constants -fM {M = 0, 1, 2, ... ) such 
that 

(4.5) < 7A/(1 + |z|)-^^e«|i"-l (2 e C^). 
Now note that 

/ + , <i + — ^ — r<i + VN (zec^,«;er^). 

1 + |z + w| - l + \z + w\ - ^ ' 

Since in addition |Im (z + u;)| < |Imz| + \fN for z £ and w € , we conclude 
from (|4.4I) that h also satisfies estimates as in (|4.5|l . i.e., e Tis^. Hence !F~^{h) is 
in 'D{Bji), and in fact T^^h e T>{Bii)'^'^'^^ since is 0(A^)-invariant. This implies 
that / e VoiBji), as required. □ 

This concludes the proof of Proposition 14.61 

We can now prove the Paley- Wiener theorem of this section by putting the pieces 
together. 

Theorem 4.10 (Paley- Wiener theorem, first version). Let G be a finite reflec- 
tion group and suppose that Kek > 0. Then the Dunkl transform D^. is a linear 
isomorphism between 'D{Bfi) and 'Hbrj for all R> 0. 

Proof. In view of the inversion theorem, all that needs to be done is to show that 
Ekg € X'(Bfl) for aU g e Hbr- To this end, start by noting that EkD^f e 'D{Bii) if 
/ € 'D{Bf{)^^^\ as an immediate consequence of Proposition H31 and Theorem 14. 41 
But then we also have EkDof £ 'D{Br) for / £ VoiBji), since Ek o Dq o — 
T^{k) o Ek o Dq. From Proposition 14.61 and continuity we can then conclude that 
EkDof £ 'D{Br) for / £ T>{Bji). By the Paley-Wiener theorem for Z?o, we see 
that Ekg £ 'D{Bb) for all g £ Hbr, as required. □ 

4.2. The case of arbitrary G and strictly positive integral k. If all ka are 

strictly positive integers. Conjecture 14.11 can be established from the correspond- 
ing theorem for the Fourier transform, by showing that Expg is then in fact an 
elementary function. This can be seen by starting from Heckman's work on shift 
operators \T^ . 
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We introduce some notation. Let R — IJ,™ Si be the decomposition of R into 
G-orbits, and put 5"^,+ = Si Ci R+. Define tt^ = HaGSi + ^* ' ^^^^ ^'^^ denote 
restriction to the G-invariant functions. Then we have the following results 
valid for p (iV'^: 

(4.6) Res (Af^/r^^(fc)) Res (Tp(fc)) = Res {Tp{k + I5.)) Res (M-/T^.(fc)) , 

(4.7) Res (r^^(A:)M^J Res {Tp{k + IgJ) ^ Res {Tp{k)) Res {T^^(k)M^,) , 

where {k + IsJ^ = /cq + 1 if a £ Si and (/c + IsJq = fc^ if a <^ 5^. 
The generalized Bessel kernel Jq is defined for k E K^°s by 

(4.8) Jg(A, fc, a;) = XI ExPg(A, k, gx) 

and is (up to multiples) the unique G-invariant solution of the Bessel system 

Tp{k)f=p{X)f {per''). 

It follows immediately from (|4.t)|) . H4.7|l and the uniqueness of invariant solutions 
of the Bessel system that there exist complex constants ci 2(A, k) such that 

M-^T^^{k)JG{X,k, .) =ci(A,fc)JG(A,fc + ls., .) 

and 

T;,(fc)M^,JG(A,fc + ls,, .) =C2(A,fc)JG(A,fc, .). 
Evaluation at zero yields C2(A, k) = (tt^, ni)k, where ( . , . )fc is the bilinear form from 
Section 13 

ci(A,fc)(x„7r,)fcJG(A,fc, .)^ci{X,k)T^^{k)M^jGi\k + ls„ .) 

= T,, {k)M^M-^T^^ {k)JGiX, fc, . ) 
= Trf (A) Jg (A, fc, .). 

Hence 

(4.9) 7r2(A)JG(A,fc + Is,, .) = {7T,,7T,)kM-^T^^{k)JG{X,k, .). 

Suppose then that the (fixed) value of k on Si is fc^, a strictly positive integer. 
Repeated application of 14. 9|) yields 

(4.10) 

771 ki 

wk{x)jG{x,k, .)= n n(^*''^*)("-i)is, fejis/ 

i—l n— 1 

4=1 n=l y j<'i J I I gGG 

We conclude from H4.10|l that there exist functions Pg{X, x) on oc x a^^ (we suppress 
the dependence on our fixed k) which are polynomials in A with rational functions 
in X as coefficients (with poles with respect to x along the singular hyperplanes, if 
any) such that 

Wk{X)JG{X,k,x) = Xpg(A,x)e(»^'") (A G oc, x e a'^s), 
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where as usual a'°^ = {x E a \ {a,x) ^ Va G R} and a^^ is defined similarly. 
Now the first operator acting in H4.1()|l is TTr^iO), showing that the pg are divisible 
by 7ri(A). Since the numbering of the Si is arbitrary we see that the pg are in fact 
divisible by 7r(A) = Jlfc^i '^iW- Thus 

(4.11) Wk{\)JG{X,k,x)^7T{X)Y,p'giK^)e^'^-''^ (A e ac, X G a'°s), 

see 

where the p'g are of the same form as the pg . It is here that we have used that the 
ka are strictly positive integers, in order to obtain the factor 7r(A) in the right hand 
side of H4.11|l which is needed below. 

There is a similar expression for Expg , which we obtain using a result of Opdam 
describing how Exp^ may be obtained from Jq- According to [loc.cit., 6.4-6.6] 
there exists a rational function Q{X,x) on oc x ac such that: 

• (A, a;) 1-^ 7r(A)Q(A,x) is a polynomial on ac x ac; 

• TQ(A..)(fc)^G(A,fc, .) = Expg(A,fc, .) (A G ac*5, x G ac). 

Using this and (|4.11() we see that, for A G a^^ and x G a'^^^^ 

(4.12) Wki\)Expa{\k,x)=TQ(^^,,-^Wk{\)JGi\,k, .) 

= TQ(,,.)^(A)5]p;(A,x)e(''^--) 

geG 

= Ep;'(A,a:)e(«^^-), 
see 

where again the p'g are of the same form as the pg. But then 1)4.12(1 is by continuity 
in fact valid for A G ac (rather than A G a^'^^) and x G a''''^. This observation allows 
us to establish Conjecture 14. II for strictly positive integral multiplicities. 

Theorem 4.11 (Paley- Wiener theorem, second version). Let G be a finite re- 
flection group and suppose that the ka are all strictly positive integers. If S is a 
non-empty G-invariant compact convex subset of a, then Dk is a linear isomorphism 
between 'D(S) andHs- 

Proof. Suppose / G Tis, x a, x ^ S . In view of the inversion theorem, the only 
thing that remains to be proved is: 



(4.13) / /(A)ExpG.(iA,fc,a;)wfc(A)dA = 0. 

J a 

If a; G a''''^, then (|4.12() allows us to write this integral as 

E / f{X)p'^{tX,x)e^<^^-^-'^Ux, 

where the p'g are polynomials in A with rational functions in x as coefficients (with 
poles with respect to x along the singular hyperplanes, if any). Hence the geometri- 
cal form of the Paley- Wiener theorem for the Fourier transform 20, Theorem 7.3.1] 
shows that each of the summands is zero. Then ((4.13|l holds for all x ^ 5* by con- 
tinuity. □ 
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4.3. The case where G is a Weyl group and Re A; > 0. In the case where G 
is a Weyl group and Re A; > 0, a Paley- Wiener theorem for intersections of convex 
hulls of orbits can be obtained by a limit transition from the results of Opdani on 
Cherednik operators for A: > j27| . If every non-empty G- invariant compact convex 
set were the intersection of the convex hulls of orbits, then this would establish 
Coni ect ure 14.11 for Weyl groups. However, this intersection property does not hold, 
as the two-dimensional example of the closed unit ball and Ai x Ai shows. Thus, 
Conjecture 14.11 remains open even for Weyl groups, and this example also shows 
that in this context neither one of the Theorems 14 . 1 01 above and 14.151 below implies 
the other. 

We start by collecting the relevant results from 27 . 

Suppose that R is an integral root system, and let fc : i? i-^- C be a multiplicity 
function. We will assume that R is reduced, which is sufficient for our purposes. 
Choose a system i?+ of positive roots, and define the Cherednik operator D^, for 
C G ac, as 

D^{k)=d^+ A;a(«,e)M^_g-„*wi(l-r„)-(p(/c),0, 



where p{k) = i X]aefl+ ^aOi. The Cherednik operators are not equivariant, and 
their definition is dependent on the choice of R-\-. For fixed multiplicity they form 
a commutative family, and together with G they then generate an algebra of op- 
erators which is an isomorphic copy of the graded Hecke algebra, corresponding to 
the choice of R+ and A;, as it is defined by Lusztig [23 ■ We refer to |3] for this 
isomorphism, or to [2^ . Aside, we remark that the D^{k) certainly depend on the 
length of the roots, in contrast to the Dunkl operators, but it will become apparent 
that this is innocent for our purposes. 

The Dunkl kernel Exp^ can be obtained from the eigenfunctions of the Cherednik 
operators by taking a suitable limit. To see this, we recall a weakened version of 
some of Opdam's results [37| c.f. Theorem 3.15 and Proposition 6.1]. 

Theorem 4.12. For all k > there exists an open neighborhood U o/ G a and a 
holomorphic function Cq{ . , A;, . ) : ac x (a + iU) ^ C with the following properties: 

(1) Gg(A, A:, 0) = 1 for all X € ac; 

(2) i?5(fc)GG(A,A;, .) = (A,OCG(A,fc, .) for all X, ^ e ac; 

(3) if X Cz ac, and z = x + iy with x € a and y d U , then 

(4 14) |Gg(A k z) < ■\/|G|e^ mingec (s Im A,K)+maXgeG (g p(fc).J/)+maXgeG (s Re \,x) 

The estimate in (|4.14ll enables us to prove the following limit transition. 

Theorem 4.13. Let k > be fixed, and suppose that S C ac x <^c "is a non-empty 
compact subset. Then, as e — > through the complex numbers, we have 

(4.15) lim GG(e""^A, k, ez) = Expq(A, fc, z), 

uniformly for (A, z) e S. Here it is understood that |e| is taken sufficiently small, 
so that GG(e~^A, k, ez) is defined for all (A, z) G S. 

Proof. Suppose, to the contrary, that there exist 77 > 0, a non-empty compact set 
5 C ac X ac, and a complex sequence e„ ^ such that, for all n, 

(4.16) sup |GG(e;;^A, A:,e„z)-ExpQ(A,fc,z)| >?7. 
(A,2)es' 
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Take an open neighborhood of S of the form Vi x V2 , with V\^Vi C ac open baUs 
centered at the origin, and define 0„ : Vi x V2 i-^ C by 0n(A, z) — Cai^n^^, k, e„z), 
discarding a finite number of e„ if necessary. From ()4.14|) one sees that the sequence 
{4>n}^=i is uniformly bounded on Vi x ¥2- By Montel's theorem we may therefore 
assume, passing to a subsequence if necessary, that the 0„ converge uniformly on 
compact subsets of Vi x V2 to a holomorphic function (poc on Vi x V2 . 

Now a small computation shows, for fixed A € Vi and for arbitrary ^ e ac, that 
as functions on V2 we have 



Passing to the limit one obtains 



(1 - r„) } 0„(A, . ) = {(C, A) + 6„(p(fc), 0} M\ ■ )• 



(a, .) 



This is the defining equation for the Dunkl kernel, which has a local solution 
space spanned by Expq(A, fc, .). Since obviously 0oo(A,O) = 1, we conclude that 
(/)oo(A, z) — Expq(A, k, z) for all (A, z) G Vi x V2. But this implies that the (pn con- 
verge uniformly to Expq{ . ,k, . ) on the compact set S C Vi x V2 after all, which 
contradicts H4.16|l . □ 

In the end, we will only need the special case where A € za of the following 
corollary. Note that the function involved is defined for all e under consideration. 

Corollary 4.14. Let fc > 0. Then, as e through the real numbers, we have 
lim^^^o C'gIc^^A, fc, ex) = Expq(A, fc, x) for all A £ ac and all x G a. 

We will now invoke Opdam's Paley- Wiener theorem. To this end, fix a; e a, let 
co(G' • x) denote the convex hull of the orbit of x, and fix y ^ co{G ■ x). Suppose 
that / € 'Hco(G x)- Then, if fcc > for all a, the Paley- Wiener theorem for the 
Cherednik transform |27l Theorem 8.6.(2)] implies that 



/(A)CG(*A,fc,zj) n 



1 - 



ka{a, a) 
2i{X,a) 



n 



/ 2i(A,a) 



ka 



r 



{ 2»(A,a) \ 



dX = 0, 



where the integrand should be read as for all A S a'^'"s_ Now note, for real e 7^ 0, 
that the map A 1— s- /(eA) is an element of 'Hco(G ex)- Since ey ^ co(G • ex), we have 



/ f{e\)CG{i\k,ey) [] 



ka{a, ct) 
2i{X,a) 



After a change of variables it follows that 
(4.17) 



/ f{X)CG{te-^X,k,ey) H (l - 



eka{a, a) 
2i{X,a) 



n 



n 



r 



2i{\,a) 
(a, a) 



kr 



/ 2ie~^(A,a) 
V (a. a) 



dX = 0. 



/ 2i£-i(A,a)\ 



dX = 0, 
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for all e > 0. Now it follows from a well-known result for the Gamma function 

p. 151] that, for each a e K, there exist constants Mi{a) , M2{a) , Mi{a) , M2{a) > 

such that, for all s e R, 

r{is + a) 



(4.18) 



(4.19) 



V is/ 



r{is) 

a \ T{is + a) 



< A/i(a)|sr + M2(a) 



<Mi(a)|.sr + M2(a) 



Furthermore, for all s, a G K, s 7^ 0, one has 
(4.20) .an^e-'s + a) 



lim 



r(ie-is) 

From (|4.18|) and H4.19|l one sees that the dominated convergence theorem applies 
in |1T7| as e i 0, since / G 5(a) and \CG{ie^'^X ,k,ey )\ < y/\G\ by glU). In the 
limit one thus obtains from (|4.2()|l and Corollary 14. 141 that 



(4.21) 



/(A)Expg(iA, k, y)wk{X) dX = 0, 



for fc > 0, a; e a, / e Hco(G x); and y ^ co{G ■ x). 
It is now an easy matter to prove the following. 

Theorem 4.15 (Paley-Wiener theorem, third version). Let G be a Weyl group 
and suppose that Re A: > 0. If S G a is an intersection of convex hulls of G- orbits, 
then the Dunkl transform is a linear isomorphism between 'D{S) and Us- 

Proof. As in the proof of Theorem 14.111 in view of the inversion theorem all that 
needs to be done is to prove that Ek : Hs S{a) has its image contained in ^{S). 
To this end, first assume that S is the convex hull of one orbit and that fc > 0. 
In that case, (14.2111 is just the required result. But then 14.21|1 actually holds for 
Refc > by analytic continuation in fc, and the case Refc > follows from this 
again by continuity. The result for an intersection of convex hulls of orbits follows 
trivially from the result for the convex hull of one orbit. □ 

Remark 4.16. The type of limit transition above has also been introduced by 
Ben Said and 0rsted The approach in [loc.cit.] starts from the invariant case, 
using shift operators, and then proceeds to the general case. The use of shift 
operators thus restricts the validity of the proofs to integral multiplicities. The 
approach to this transition in the present paper appears to be somewhat simpler, 
since Corollarv l4.14l is seen to hold directly in the general case and for more general 
multiplicities. 

5. Extending the intertwining operator 

In this section we return to the general situation of an arbitrary reflection group 
and Refc > 0. With the aid of the Paley-Wiener theorems, we will define a lin- 
ear isomorphism Vk : C°°{a) ^ C°°(a), such that mk)Vk = Vkd^ (^ G a), and 
which extends the intertwiner operator as it has originally been constructed for 
polynomials by Dunkl for fc > j7|. The polynomial case for general regular fc 
was considered in |13j and an extension of the intertwiner operator to an algebra 
of real analytic functions was constructed in for fc > 0. The extension to C°°{a) 
for Re fc > as in this section was first established in |22] . The case where fc > 
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and J2a£R+ ka > was later on also considered in 38 , using Rosler's representing 
measures which are presently only known to exist when A; > 0. 

Recall that the symmetry S, defined by Sf{x) = f{—x), relates Ek and by 
SDk = DkS = Ek ^ Lemma 4.3]. The Paley- Wiener theorem QUI therefore 
implies that DkEo = SE^Eq maps I?(a) into itself. This map is easily seen to 
be continuous in the standard inductive limit topology (e.g., as a consequence of 
the closed graph theorem) and it is actually a homeomorphism since the inverse is 
DoEk- Thus the transpose {DkEoY : 25'(a) T^'i*^) is a homeomorphism in the 
weak*-topology, leaving S'{a) invariant as a consequence of part (5) in Theorem l2.2l 

For the formulation of the following theorem, we let B'j^ denote the open ball of 
radius R, centered at the origin. 

Theorem 5.1. Let G be a finite reflection group and suppose that Refc > 0. The 
map Wk ■■ C°°(a) ^ ^'(a), defined by 

Wkf = ^{DkEoYifwk), 

Ck 

is actually a linear automorphism of C°°{a). Let Vk '■ C°°(o) C°°(a) be its 
inverse. Then both Wk and Vk commute with the G-action and the following hold: 

(1) WkT^{k)=d^Wk (ee a). 

(2) Wkl = 1. 

(3) WkVn^Vn- 

(4) (a) If f & C°°{a) vanishes on B'j^, then Wkf vanishes on B'j^. 

(b) If U is an open G-invariant convex subset of a, and f G C°°{a) van- 
ishes on U , then Vkf vanishes on U . 

(5) If f ^ C°°{a) Li{a,\wk{x)\dx), Dkf ^ Li{a,\wk{x)\dx), and x ^ a, then 

Wkfix)^-Eo{wkDkf){x) = - [ DkfiX)cxp{i{X,x))wkiX)dX. 

Ck Ck J a 

(6) If f e C°°{a) n Li{(x,dx), Dof G Li{a,dx), and x <E a, then 

Vkfix) = ^Ekiw^'Dof) = - f Z?o/(A)ExpG(ix,A;,A)dA. 

Co Co Ja 

Proof. As a preparation, we recall from [2.U Lemma 4.13] that J^{Dkf)gwkdx = 
Ja f{Dkg)wk dx for f,g <E Li{a, \wk{x)\dx), and similarly for Ek. 

Suppose T e V'{a) and suppT n = 0. Then supp (Dfc-Bo)*^ n B^ = 0. 
Indeed, let V e Then {(DkEoYT^^jj) = {T^DkEo^P) = {T.EkD^^) = by 

the Paley-Wiener theorem 14. 101 

Now let / e C°°{a). Fix i? > and choose (j) € V{a) such that (/) = 1 on 
B'j^. Then supp {4>fwk — fwk) H B'j^ = 0, hence by the above we have for arbitrary 
4, G ViB'j,): 

(5.1) {(DkEoYifwk), V) - {{DkEoYWwk), ^) 

ip{x)Eo {Dk{(pf)wk) (x) dx. 

Note that Dk{(l)f) G 5(a), so Eq {Dk{(l)f)wk) is smooth. Since the left hand side 
in (|5.1() does not depend on cj), the restriction of Eq (ffe ((/>/) u'fc) to B'j^ is appar- 
ently also independent of the choice of 4>- Hence we can unambiguously define a 
smooth function representing the distribution (DkEoYifwk), by choosing for a; G a 
any B'j^ such that x G B'j^, any (p G T>{a) such that = 1 on B'j^, and putting 
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{{DkEoYifwk)) (x) = Eo {Dk{(t)f)wk) [x). This shows that Wk maps C°°(o) into 
itself. It follows from the remarks before the statement of the theorem that Wk is 
injective. 

In order to see that Wk is also surjective we consider {DoEkY- For {DoEkY a 
geometrically more precise property with respect to supports can be proved than 
for (DkEoY- To be precise: if U is an open G-invariant convex subset of a, and 
T € V'{a) is such that suppT n J7 = 0, then supp {D^EkYT n C/ = 0. Indeed, if 
ij} e V{U), then {{DoEkYT, ijj) = (T, D^Eki^) = (T, E^Dkij) = by the geometrical 
form of the Paley- Wiener theorem for the ordinary Fourier transform — note that 
Dkip is of Paley- Wiener type corresponding to the G-invariant compact convex set 
co(G • suppV') C U . Let R> and / e G°°(o) and choose (p as above. Taking the 
open ball B'j^ as U, we then compute as follows for arbitrary ip £ 'D{B'j^): 

(5.2) {{D.EkYf, ^) = {{DoEkY {(t^f). ^) 

= f i?o(0/)(A)(i?feV)(A)dA 



Cfc^ I 'Do(0/)(A)|y ^{x)ExpQ{ix,k,X)wk{x)dxj dX 
Ck^ / Do{<pf){X)F,xpQ{ix, k, X) dX> iIj{x) Wk{x) dx 



Now the function : x ^ '^k^ I ^o(0/)(A)ExpQ(za;, fc, A) dA is smooth on a 
since the derivatives of ExpQ(j . , fc, A) are of polynomial growth in A 21 , Corollary 
3.7] and Z?o(0/) S '5(a). The g^j, patch together, as before, to a smooth function 
5 on a which satisfies {D^EkY f = gwk by construction. Taking inverses we find 
{DkEoY{9'Wk) — /, showing that Wk is surjective, as required. 

It is easy to check that Wk and hence its inverse Vk commute with the G-action, 
using an invariant function in the construction, and noting that Dk and Eq have 
this property. 

Continuing, we define the Dunkl operator r^'(fc) on I?'(a) by {T^{k)T,iJ;) = 
-{T, T^{k)^). We have the compatibility T'^{k){fwk) = {T^{k)f)wk for / e G°°(a). 
Hence 

{{DkEoY{{Ti{k)f)wk).^) = {{T^{k)J)wk,DkEoi^) 

= {T^{k){fwk),DkEo4') 
= -{fwk^DkEod^ip) 

= {d^{{DkEoY{fwk)),^), 



proving part (1). 

For part (2) and part (3) we consider : S'{a) i-^ S'{a). By the inversion 
theorem we have iP^Wfc = CkS, hence {DkEoYwk = CkEgS = CkC^ , proving part (2). 
Furthermore, one easily verifies the relation Dl{M^.T) = iT^{k)DlT for T € S'{a) 
and ^ e a. Thus for p e Vn we have D\{pwk) — Cki'^T^{k)5. By the homogeneity 
of the T^{k), there exists a differential operator D with constant coefficients and of 
homogeneous degree n such that Cki"Tp{k)S = DS. Hence {DkEoY{p'Wk) = EqD6 
is in Vn, as stated in part (3). 
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Part (4) follows from the properties regarding the supports of distributions which 
we have used above. 

The parts (5) and (6) follow by performing the computations in H5.1|l and (|5.2|l 
with 0=1, which is validated by the integrability conditions. □ 

Remark 5.2. 

(1) It is easily seen that the validity of Coniecture l4.1l would imply that the open 
ball in part (4) (a) for Wk could be replaced by an open G-invariant convex 
set, as for Vk in part (4)(b). In fact, if all multiplicities are strictly positive 
integers, then we already know that this stronger statement actually holds, 
in view of Theorem 14.111 Likewise, if the groups is a Weyl group, then we 
known that this stronger statement holds for the interior of an intersection 
of convex hulls of orbits, in view of Theorem 14. 151 

(2) For fc > 0, part (4)(b) also follows from the properties of Rosler's repre- 
senting measures, cf. (3Bj . 

6. Connection with the Cartan motion group 

If G is a Weyl group W, then, for certain multiplicities, the M^-invariant part of 
the theory for Dunkl operators has an interpretation in terms of the Cartan motion 
group, as we will explain in this section. The starting point is Heckman's result 

1 / A \ " 

(6.1) rp(/c) = - (^ad^j Mp, 

for p G Vn, cf. Section 131 A double application of this result enables us to give 
an explicit description of the radial parts in terms of Dunkl operators. From this 
description we can then conclude that the generalized Bessel functions coincide with 
the restriction of the spherical functions to a. This identification, in turn, enables 
us to identify the restriction to the invariants of the operators Wk and Vk from 
Theorem 15. II in terms of (the flat analogue of) the Abel transform. Furthermore, 
we show that in certain cases shift operators can be used to essentially invert the 
Abel transform by an invariant differential operator. 

Establishing terminology, let G he a connected non-compact semisimple Lie 
group with finite center, maximal compact subgroup JC and corresponding Car- 
tan decomposition g = 6 © p of the Lie algebra. The group Go = !^ ^ P acts on the 
flat symmetric space Go/^^ ~ p as isometrics for the Killing form. The group of 
isometrics of p thus obtained is known as the Cartan motion group. 

Choose a maximal abelian subspace a C p. Then there is a restriction iso- 
morphism ReSg : C°°{p)^ i-^ G°°(a)^, with analogues for compactly supported 
smooth invariant functions and for invariant polynomials. Let E be the restricted 
roots with multiphcities (a G E). We consider E to be a subset of a by means 
of the Killing form ( . , . ) and we let W be the Weyl group, acting on a. 

The spherical function ip\ (A £ ac) on the symmetric space p satisfies the equa- 
tions d{p)ip\ ~ p{X)ijj\ for all p in the algebra >5'(p)'^ of /C-invariant polynomials on 
p, where as usual d{p) is the constant coefficient differential operator correspond- 
ing to p. Furthermore, '(/'a(O) — 1. By (1^ Corollary 2.3, p. 402], tpx is the unique 
/C-invariant function on p with these two properties. 

We let i? C a be the normalized root system of the finite reflection group W (so 
some roots in a component BCn of E will coincide in R). Choose the multiplicity 
function k : R i-^ R a.s ka — ^/'^J^sem.anY: ™/3- With these multiplicities, we define 
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the corresponding operators Tp{k) (acting on functions on a) for the finite reflection 
group W. The crucial observation, with Rad denoting radial part, is the following: 



(6.2) Rad(a(p))/ = TR,,Pp(fc)/ (p e Sip)'^, f e C°°(a)^) . 

Before proving this relation, let us note that it shows how the algebra of radial 
parts can be obtained from Dunkl operators, as follows. One computes the operator 
Tq(fc) for each polynomial g in a set of fundamental invariants, and then restricts 
it to the invariant functions on a. This restriction acts as a differential operator 
(a moment's thought will make this clear), and the differential operators which are 
thus obtained generate the algebra of radial parts. 

Turning to the proof of l6.2l we first recall |B] that 

Afe = Ao + 2 J2 kc.M(^^,yi {da, - M(„.)-i(l - r„)} . 

Let Ap denote the ordinary Laplacian on p. By the choice of fc, the expression for 
Rad Ap in 18, Proposition 3.13, p. 270], and the above expression for A^, we have 
RadAp/ = Afc/for/eC-(a)^. 

Now let p £ S{p)'^ be homogeneous of degree n and suppose / £ C°°{a)^ . We 
apply (|6.1|) first on p (with k ~ 0) and then on a (for our particular choice of k) to 
prove (|6.2|l : 

Rad(a(p))/ = ^Rad (^ad Mp^ f 

If/. Rad A 



1 . A 



ad^-^ (RadAfp)W 



After having identified the radial parts in terms of Dunkl operators, it is now 
easy to show how the Bessel kernel Jw-, i-e., the W^-invariant component of Exp^y 
as in (|4.8|l . is related to the restrictions of the spherical functions to o. Indeed, if 
A £ oc, then we note that Jvk(A, k, . ) has an extension JJ^'(A, fc, . ) to a smooth /C- 
invariant function on p. This extension is evidently equal to 1 in 0, and satisfies the 
equations ^{p)J^;*^{X, k, .) = p{X)Jy^*'{X, k, . ) for all p £ S{p)'^, as a consequence 
of 1)6.2(1 and the very definition of radial parts. By the uniqueness mentioned above, 
we conclude that J{^*(A, k, x) = ip\{x) for X £ oc and x £ a. 

In this context of the Cartan motion group, we will now proceed to identify 
the restrictions to the invariants C°°{a)'^ of the linear automorphisms Wk,Vk : 
C°°(a) i-> C°°(o) from Theorem O RecaU [THl p. 467] that 

(6.3) / f{x)dx = (27r)^c-i / Res ^/(x)^^ da; (/ £ C^{pf). 

The constant is determined by considering the Gaussian. Since we have identified 
the spherical functions as Bessel functions, we see from this integral formula that 
the normalized spherical transform J^, defined by 

Tf{X) = (27r)-^ / /(x)V'-.A(x)dx (A e a, / e C^{pf), 
Jp 



PALEY-WIENER THEOREMS FOR THE DUNKL TRANSFORM 



23 



factors &fi J- — IJ/cRes^. From this relation one sees that the inversion, Plancherel 
and Paley- Wiener theorems for the spherical transform follow from the correspond- 
ing theorems for D^, when specialized to the invariants (the first two admittedly 
being almost trivial for the Cartan motion group). 

Let q be the orthoplement of a in p and define the (flat analogue of) the Abel 
transform: 

A}{x) = I f{x + q) dq if e CT'ipf, X e a). 

Jq 

Since the spherical transform can also be written as 

Tf{X) = (2^)-^ / /(x)e-*(^'-) dx (A G a, / G C^{pf), 
Jp 

Fubini's theorem yields an additional factorization of JF, namely T — (2tt) DqA. 
The relation (27r)-^DoA/ = D^RcsP/ for / G C^ip)'^ and the Paley-Wiener 
theorems for Dq and Dk together then show that the Abel transform establishes 
a linear isomorphism between C^(p)'^ and C^{a)^. It is easily verified that 
A : C^(p)'^ 1-^ C^(a)'^ is in fact a linear homeomorphism, so that the isomor- 
phisms A* : V'{a)^ V'ip)'^ and (A"!)* : V'{p)'^ ^ V'{a)^ are defined. 

Recall the symmetry S from Section^] defined by Sf{x) = Sf{—x), and note 
that, as maps from C^(a)^ into itself, we have 

DkEo = DkSDo = {2t:)^ DkSTA-^ - DkSDkReslA-^ 

= {271)^ DkEkKeslA-^ = (27r)'^ResP^"^ 

Let ExfP : C°°(a)^ ^ C°°(p)'= be the inverse of Res P. Using we find, for 

/ G C°°(a)^ and ip G C^{a)^: 

{Wkl.i>). = {2Ti)'^c-\{DkEaY{fwk),^)a 
= {2n)^cl\fwk,DkEo4,)^ 

= (27r)^c^i(/w;fc,ResPA-V)a 

= (ExtP/,A-V)p 

= ((A-i)*ExtP/,V)„. 

We conclude that Wfe = (A"i)*ExtP, as maps with domain C°°(o)^. For the 
right hand side, we know a priori only that it maps C°°(o)^ into I?'(a)^, but 
this equality and Theorem 15.11 show that it is actually a linear automorphism of 
C°°(a)'^. It follows from this that (A~^)*, which a priori is only known to map 
C°°(p)'*" C T)' (pY~ into I?'(a)'^, establishes in fact a linear isomorphism between 
C°°(p)'^ and C°°(o)^ . This enables us to identify the restriction of Dunkl's inter- 
twiner to C°°(a)^ as Vfc = Res^A*, where the fact that A* : C°°(a)^ ^ C^{pf- 
is a linear isomorphism is then implicitly used in the interpretation of the right 
hand side. 

Remark 6.1. For certain multiplicities of the restricted roots, the results on shift 
operators can be used to essentially invert the Abel transform, as follows. Assume 
that ka = l/4X]/36Eans ™/3 integer for all a £ R. Then l|4.10|l implies 

that there exists a W^-invariant differential operator D of order 1/4 X^qgs '^a , with 
poles along the singular hyperplanes only, such that Ekg — DE^g for all rapidly 
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decreasing g e C°°(a)^. Therefore, if / e C^{af , we find that Rcs^A-^ f = 



This shows that the Abel transform is for such muhiphcities essentially inverted by 
the H^-invariant differential operator (27r) ^ D. 

Remark 6.2. In this section it has become clear that the theory of spherical 
functions for the Cartan motion group is intimately connected with the invariant 
part of the theory of Dunkl operators. It can, however, not be excluded that 
the general theory of these operators is also relevant in this context, for conceptual 
reasons which have gone unnoticed sofar. The author at least finds it hard to believe 
that, e.g., the description of the algebra of radial parts should not be the reflection 
of some more direct connection. There are, in effect, two additional indications 
that such a connection may exist. 

The first of these is given by the estimates on the spherical functions ■0a- One 
has the integral representation 



for A e ocand x E a. The Kostant convexity theorem 18, Theorem 10.2, p. 473] 
then yields the estimate jV'Ala^)! < exp(max^gn/ Re {wX, x)). But alternatively, this 
also follows from the estimate for Expi^ in Section |2] and the identification of the 
spherical functions as generalized Bessel functions, as explained in this section. This 
natural occurrence of Exp^^ points to relevance of the original Dunkl operators in 
the Cartan motion group case. 

The second indication consists of Torossian's application of Dunkl operators in 
the inversion of the Chevalley restriction isomorphism between the /C-invariant poly- 
nomials on p and the VF- invariant polynomials on a [23 • Here the original Dunkl 
operators make an appearance, not just the operators corresponding to invariant 
polynomials. 

Remark 6.3. When the results in this section are combined with similar results 
for the Cherednik operators in Section 14.31 and with Theorem 14.131 one obtains 
a limit transition from the spherical functions for a Riemannian symmetric space 
of the non-compact type to the spherical functions for the corresponding Cartan 
motion group. This has previously been proved in a direct fashion by Ben Said and 
0rsted in |2|. This limit transition enables one to obtain explicit formulas for the 
spherical functions for the Cartan motion group. 
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